Skip to content

[2/4] Add DeviceStatsMonitor callback #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 55 commits into from
Closed

[2/4] Add DeviceStatsMonitor callback #2

wants to merge 55 commits into from

Conversation

daniellepintz
Copy link
Owner

@daniellepintz daniellepintz commented Sep 17, 2021 •
edited
Loading

What does this PR do?

Adds DeviceStatsMonitor callback which calls the get_device_stats function in Accelerator interface. DeviceStatsMonitor will replace GPUStatsMonitor and XLAStatsMonitor

Fixes Lightning-AI#9032

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

@daniellepintz daniellepintz mentioned this pull request Sep 18, 2021
Merged
12 tasks
@daniellepintz daniellepintz changed the title [2/n] Add DeviceStatsMonitor callback [2/4] Add DeviceStatsMonitor callback Sep 18, 2021
daniellepintz and others added 22 commits September 18, 2021 10:05
@daniellepintz @awaelchli @tchaton
Deprecate TrainerProperties Mixin and move property definitions direc…
290398f 
…tly into `trainer.py` (Lightning-AI#9495)

Co-authored-by: Adrian Wälchli <[email protected]>
Co-authored-by: thomas chaton <[email protected]>
@hankyul2
Fix typo in LightningModule.configure_callbacks() (Lightning-AI#9591)
c66d30a
@daniellepintz
Put back initialization of properties in trainer (Lightning-AI#9594)
381343a
@LaserBit
Fix missing url (Lightning-AI#9602)
2e17b47
@jjenniferdai @pre-commit-ci
@ananthsub @awaelchli @tchaton @Borda
Deprecate LightningLoggerBase.close (Lightning-AI#9422)
cc77367 
* deprecate loggerbase.close

* deprecate warning

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add to changelog

* fix import

* fix import alphabetize

* spacing?

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* copy-paste avoid pre-commit.ci?

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* literally match the other comment

* unindent

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* suggest finalize instead of save

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update tests/loggers/test_base.py

* format but to be formatted

* Update pytorch_lightning/loggers/base.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update pytorch_lightning/loggers/base.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update pytorch_lightning/loggers/base.py

Co-authored-by: Jirka Borovec <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: ananthsub <[email protected]>
Co-authored-by: Adrian Wälchli <[email protected]>
Co-authored-by: thomas chaton <[email protected]>
Co-authored-by: Jirka Borovec <[email protected]>
@carmocca
[CLI] Fix registry decorator return value (Lightning-AI#9587)
cd8cb60
@carmocca
[CLI] Avoid warning when configure_optimizers will not be overridden (
61b4e33 
Lightning-AI#9583)
@daniellepintz
Prune DeprecatedTrainerAttributes (Lightning-AI#9598)
74c3536
@carmocca
Update versions after recent PyTorch releases (Lightning-AI#9623)
d022f6f
@Benjamin-Etheredge
Fixing order of operations bug in Qnet (Lightning-AI#9621)
73e53e5
@ananthsub
Fix gradient accumulation for ShardedDataParallel (Lightning-AI#9122)
a71be50 
* Fix gradient accumulation for `ShardedDataParallel`

* Update changelog

* Update pytorch_lightning/plugins/training_type/sharded.py

* add test

* Update test_sharded_plugin.py

* Update test_sharded_plugin.py

* Update test_sharded_plugin.py
@danielykim
Fix broken links to PyTorch Lightning Bolts (Lightning-AI#9634)
e64f358
@carmocca
Fix ResultCollection._get_cache with multielement tensors (Lightnin…
8f1c855 
…g-AI#9582)
@carmocca
[CLI] Shorthand notation to instantiate models (Lightning-AI#9588)
3f7872d
Improvements for rich progress (Lightning-AI#9579)
fd2e778
@awaelchli
revert Lightning-AI#9125 / fix back-compatibility with saving hparams…
b98ce0a 
… as a whole container (Lightning-AI#9642)
@four4fish @awaelchli
fix modify _DDPSinkBackward view inplace error for pytorch nightly 1.…
37469cd 
…10 (Lightning-AI#9649)

Co-authored-by: Adrian Wälchli <[email protected]>
@awaelchli
update changelog after 1.4.8 release (Lightning-AI#9650)
4bebf82
@mauvilsa
[CLI] Add option to enable/disable config save to preserve multiple f…
eb6aa7a 
…iles structure
@carmocca
Remove InternalDebugger.track_lr_schedulers_update (Lightning-AI#9653)
bffc534
@rohitgr7
Reset metrics before each task starts (Lightning-AI#9410)
86dd318 
* reset metrics
@daniellepintz
Deprecate progress_bar_refresh_rate from Trainer constructor (Light…
491e4a2 
…ning-AI#9616)
awaelchli and others added 27 commits September 23, 2021 09:52
@awaelchli @carmocca
@pre-commit-ci @Borda
add legacy load utility (Lightning-AI#9166)
87b11fb 
Co-authored-by: Carlos Mocholí <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jirka Borovec <[email protected]>
@carmocca
Use completed over processed in reset_on_restart (Lightning-AI#…
ca5459e 
…9656)
@carmocca
Remove InternalDebugger.track_event (Lightning-AI#9654)
fd4f2f6
@carmocca
Add is_last_batch to progress tracking (Lightning-AI#9657)
8dcba38
@stancld @awaelchli
@tchaton @carmocca
Disallow invalid seed string values (Lightning-AI#8787)
568a1e0 
Co-authored-by: Adrian Wälchli <[email protected]>
Co-authored-by: thomas chaton <[email protected]>
Co-authored-by: Carlos Mocholí <[email protected]>
@carmocca
Use searchsorted over argmax (Lightning-AI#9670)
2b2537d
@ananthsub
Remove call_configure_sharded_model lifecycle property (Lightning-A…
41e3be1 
…I#9612)
@daniellepintz
Add torch v1.11.0 to the list of versions in adjust_versions.py (Ligh…
c2e3ec1 
…tning-AI#9679)
@awaelchli @carmocca
Report leaking environment variables in tests (Lightning-AI#5872)
714331b 
Co-authored-by: Adrian Wälchli <[email protected]>
Co-authored-by: Carlos Mocholi <[email protected]>
@awaelchli @pre-commit-ci
remove InternalDebugger.track_load_dataloader_call (Lightning-AI#9675)
d67aff7 
* wip

* reset _notebooks

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reset _notebooks

* testing with mock

* update test with mock

* update test

* update tests

* update test

* remove track_load_dataloader_calls

* update last test

* remove unused imports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@carmocca @awaelchli
Support skipping to validation (Lightning-AI#9681)
ce00053 
Co-authored-by: Adrian Wälchli <[email protected]>
@tchaton @carmocca
Enable DataLoader state restoration for the evaluation loop (Lightnin…
9148a13 
…g-AI#9563)

Co-authored-by: Carlos Mocholi <[email protected]>
@rohitgr7 @carmocca
Fix accumulate_grad_batches on init (Lightning-AI#9652)
8fcdcb5 
* fix accumuate_grad_batches on init

* chlog

* update error

* move to callback connector

* add test with callback

* fix tests

* Update pytorch_lightning/trainer/connectors/callback_connector.py

Co-authored-by: Carlos Mocholí <[email protected]>

* update ipu logic

* rev

* rev

* rev

* pls work

* code review

Co-authored-by: Rohit Gupta <[email protected]>
Co-authored-by: Carlos Mocholí <[email protected]>
@carmocca
Rename reset_on_epoch to reset_on_run (Lightning-AI#9658)
d02fc2b
@daniellepintz
Add enable_progress_bar to Trainer constructor (Lightning-AI#9664)
b3a5c7f
@awaelchli
move get_active_optimizers to utilities (Lightning-AI#9581)
5395ceb
@rohitgr7
Use a unique filename to save temp ckpt in tuner (Lightning-AI#9682)
a3def9d 
* unique filename

* chlog

* update tests
@jjenniferdai @pre-commit-ci @ananthsub
Optimize non-empty directory warning check in model checkpoint callba…
444b21d 
…ck (Lightning-AI#9615)

* pt1 dir empty check

* clean imports

* bring back resolve mkdir?

* original doc

* warningcache

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* cp callback after resolve

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* move global_zero check outside warn fn

Co-authored-by: ananthsub <[email protected]>

* move global_zero check outside warn fn 2

Co-authored-by: ananthsub <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: ananthsub <[email protected]>
@four4fish @awaelchli @ananthsub
Deprecate LightningDistributed and keep logic in ddp/ddpSpawn directly (
ddf6967 
Lightning-AI#9691)

* Deprecate LightningDistributed and keep logic in ddp/ddpSpawn directly

* Deprecate LightningDistributed and keep logic in ddp/ddpSpawn directly

* Deprecate LightningDistributed and keep logic in ddp/ddpSpawn directly

* Deprecate LightningDistributed and keep logic in ddp/ddpSpawn directly

* Apply suggestions from code review

Co-authored-by: Adrian Wälchli <[email protected]>

* Apply suggestions from code review

Co-authored-by: ananthsub <[email protected]>

* Deprecate LightningDistributed and keep logic in ddp/ddpSpawn directly

* Update pytorch_lightning/distributed/dist.py

Co-authored-by: ananthsub <[email protected]>

* Deprecate LightningDistributed and keep logic in ddp/ddpSpawn directly

* Apply suggestions from code review

Co-authored-by: ananthsub <[email protected]>

* Apply suggestions from code review

* Apply suggestions from code review

* Deprecate LightningDistributed and keep logic in ddp/ddpSpawn directly

Co-authored-by: Adrian Wälchli <[email protected]>
Co-authored-by: ananthsub <[email protected]>
@rohitgr7
Update warnings in TrainingTricksConnector (Lightning-AI#9595)
a4bc0ac 
* update warnings

* add tests

* comments

* Apply suggestions from code review

* Apply suggestions from code review
@ananthsub @awaelchli
@kaushikb11 @Borda @rohitgr7
Deprecate stochastic_weight_avg from the Trainer constructor (Lig…
36b9ff2 
…htning-AI#8989)

* Deprecate `stochastic_weight_avg` from the `Trainer` constructor

* Update CHANGELOG.md

* Apply suggestions from code review

Co-authored-by: Adrian Wälchli <[email protected]>

Co-authored-by: Adrian Wälchli <[email protected]>
Co-authored-by: Kaushik B <[email protected]>
Co-authored-by: Jirka Borovec <[email protected]>
Co-authored-by: Rohit Gupta <[email protected]>
@rohitgr7
Fix lr_find to generate same results on multiple calls (Lightning-A…
83d83ab 
…I#9704)

* dump global_step

* add test

* chlog
@daniellepintz
[1/4] Add get_device_stats to accelerator interface (Lightning-AI#9586)
ab06987
@daniellepintz
Merge branch 'master' of https://github.com/PyTorchLightning/pytorch-…
e83a362 
…lightning into dstats_callback
@daniellepintz
small fix
07dd196
@daniellepintz
small fix
4124f15
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
4b81dd9 
for more information, see https://pre-commit.ci
@daniellepintz
Copy link
Owner Author

Closing in favor of Lightning-AI#9712

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.